A distributed system typically contains a plurality of processors, subsystem components, instruments and other electronic devices. These may include a number of software and real-time system monitoring devices for system environmental parameters. These devices, processors, intra processor components, system software and hardware components need to be synchronized with one another in order to correlate the occurrence of any usual or unusual software or hardware events spanning over one or more independent functional units. Thus, the synchronization in time to a desired precision level affects both the design and the debugging of the distributed system before and after the hardware manufacture.
With respect to the occurrence of faults in large scale distributed systems, logging hardware and software events, isolating faults and identifying problems are some of the most difficult tasks to perform. In order to achieve these features, it is necessary to precisely order the events in terms of their occurrence through synchronized time stamps. Usually time stamps are obtained by issuing system calls to the operating system. However, this approach does not address problems such as, once a node fails, there is no guarantee that the system call will be able to obtain the time stamp successfully executed. In addition, there is a non-deterministic processing time to service the system call which makes precise time stamping difficult. Furthermore, multiple error events may get the same time stamp which prevents event ordering. Even if an ordering scheme through indirect time stamping can solve such a problem, it is time consuming to record the events, process these events and then take action for the system. Finally, if all events get the same treatment in terms of preprocessing and taking an action, the time required to take an action might be too long to prevent short term events accruing within a particular node. Hence there is a need, not only to precisely record the occurrences of faults without system intervention, but also to design a system which is able to address the long term and short term events in such a manner that the action plan is more effective. There is also a requirement to have the machine state of any distributed system properly frozen for future debugging and probing.